Systems and methods for filling occluded information for 2-d to 3-d conversion

ABSTRACT

The present invention is directed to systems and methods for processing 2-D to 3-D image conversion. The system and method includes erode filling when an object has been removed from an image. In one embodiment, a target binary t image mask of the removed object is used for comparison with a binary source mask of the image. In an iterative process, color differences between pixels in the target mask and pixels in the source mask are identified and the pixels in the target mask are changed working from the periphery of the target mask to the center of the mask. Color noise is eliminated in the filled region of the target mask and, if desired, random color noise is added to the filled target mask.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 60/894,450 filed Mar. 12, 2007 entitled “TWO-DIMENSIONAL TOTHREE-DIMENSIONAL CONVERSION”, the disclosure of which is incorporatedherein by reference and is also related to U.S. patent application Ser.No. (not yet issued) filed concurrently herewith, Attorney Docket No.69126-P007U.S. Pat. No. 1,071,2471 entitled “SYSTEM AND METHOD FOR USINGA MASK TO GENERATE A MODEL, OR A MODEL TO GENERATE A MASK, IN THECONVERSION OF TWO-DIMENSIONAL IMAGES TO THREE-DIMENSIONAL IMAGES”; U.S.patent application Ser. No. (not yet issued) filed concurrentlyherewith, Attorney Docket No. 69126-P008US-10712472 entitled “SYSTEMSAND METHODS FOR 2-D TO 3-D CONVERSION USING DEPTH ACCESS SEGMENTS TODEFINE AN OBJECT”; U.S. patent application Ser. No. (not yet issued)filed concurrently herewith, Attorney Docket No. 69126-P009U.S. Pat. No.1,071,2473 entitled “SYSTEM AND METHOD FOR USING FEATURE TRACKINGTECHNIQUES FOR THE GENERATION OF MASKS IN THE CONVERSION OFTWO-DIMENSIONAL IMAGES TO THREE-DIMENSIONAL IMAGES”; U.S. patentapplication Ser. No. (not yet issued) filed concurrently herewith,Attorney Docket No. 69126-P010US-10712474 entitled “SYSTEMS AND METHODSFOR GENERATING 3-D GEOMETRY USING POINTS FROM IMAGE SEQUENCES”; U.S.patent application Ser. No. (not yet issued) filed concurrentlyherewith, Attorney Docket No. 69126-P011US-10712476 entitled “SYSTEMSAND METHODS FOR TREATING OCCLUSIONS IN 2-D TO 3-D IMAGE CONVERSION”;U.S. patent application Ser. No. (not yet issued) filed concurrentlyherewith, Attorney Docket No. 69126-P013U.S. Pat. No. 1,071,2478entitled “SYSTEM AND METHOD FOR USING TEMPORAL FILL TECHNIQUES WITHCHANGE IN LIGHTING IN THE CONVERSION OF TWO-DIMENSIONAL IMAGES TOTHREE-DIMENSIONAL IMAGES”, U.S. patent application Ser. No. (not yetissued) filed concurrently herewith, Attorney Docket No. 69126-P014U.S.Pat. No. 1,071,2479 entitled “SYSTEMS AND METHODS FOR ALLOWING A USER TODYNAMICALLY MANIPULATE STEREOSCOPIC PARAMETERS”, and U.S. patentapplication Ser. No. (not yet issued) filed concurrently herewith,Attorney Docket No. 69126-P015U.S. Pat. No. 1,071,2480 entitled “SYSTEMSAND METHODS FOR DEPTH PEELING USING STEREOSCOPIC VARIABLES DURING THERENDERING OF 2-D TO 3-D IMAGES,” the disclosures of which areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure is directed towards two-dimensional (2-D) tothree-dimensional (3-D) conversion of images. More specifically thepresent disclosure is directed to systems and methods for replacinginformation lost during 2-D to 3-D image conversion.

BACKGROUND OF THE INVENTION

Humans perceive the world in three spatial dimensions. Unfortunately,most of the images and videos created today are 2-D in nature. If wewere able to imbue these images and videos with 3-D information, notonly would we increase their functionality, we could dramaticallyincrease our enjoyment of them as well. However, imbuing 2-D images andvideo with 3-D information often requires completely reconstructing thescene from the original 2-D data depicted. A given set of images can beused to create a model of the observer (camera/viewpoint) together withmodels of the objects in the scene (to a sufficient level of detail)enabling the generation of realistic alternate perspective images of thescene. A model of a scene thus contains the geometry and associatedimage data for the objects in the scene as well as the geometry for thecameras used to capture those images.

A number of technologies have been proposed and, in some cases,implemented to perform a conversion of one or several two dimensionalimages into one or several stereoscopic three dimensional images. Theconversion of two dimensional images into three dimensional imagesinvolves creating a pair of stereoscopic images for each threedimensional frame. The stereoscopic images can then be presented to aviewer's left and right eyes using a suitable display device. The imageinformation between respective stereoscopic images differ according tothe calculated spatial relationships between the objects in the sceneand the viewer of the scene. The difference in the image informationenables the viewer to perceive the three dimensional effect.

An example of a conversion technology is described in U.S. Pat. No.6,477,267 (the '267 patent). In the '267 patent, only selected objectswithin a given two dimensional image are processed to receive a threedimensional effect in a resulting three dimensional image. In the '267patent, an object is initially selected for such processing by outliningthe object. The selected object is assigned a “depth” value that isrepresentative of the relative distance of the object from the viewer. Alateral displacement of the selected object is performed for each imageof a stereoscopic pair of images that depends upon the assigned depthvalue. Essentially, a “cut-and-paste” operation occurs to create thethree dimensional effect. The simple displacement of the object createsa gap or blank region in the object's background. The system disclosedin the '267 patent compensates for the gap by “stretching” the object'sbackground to fill the blank region.

The '267 patent is associated with a number of limitations.Specifically, the stretching operations cause distortion of the objectbeing stretched. The distortion needs to be minimized to reduce visualanomalies. The amount of stretching also corresponds to the disparity orparallax between an object and its background and is a function of theirrelative distances from the observer. Thus, the relative distances ofinteracting objects must be kept small.

Another example of a conversion technology is described in U.S. Pat. No.6,466,205 (the '205 patent). In the '205 patent, a sequence of videoframes is processed to select objects and to create “cells” or “mattes”of selected objects that substantially only include informationpertaining to their respective objects. A partial occlusion of aselected object by another object in a given frame is addressed bytemporally searching through the sequence of video frames to identifyother frames in which the same portion of the first object is notoccluded. Accordingly, a cell may be created for the full object eventhough the full object does not appear in any single frame. Theadvantage of such processing is that gaps or blank regions do not appearwhen objects are displaced in order to provide a three dimensionaleffect. Specifically, a portion of the background or other object thatwould be blank may be filled with graphical information obtained fromother frames in the temporal sequence. Accordingly, the rendering of thethree dimensional images may occur in an advantageous manner.

In reconstructing these scenes, features in the 2-D images, such asedges of objects, often need to be identified, extracted and theirpositions ascertained relative to the camera. Differences in the 3-Dpositions of various object features, coupled with differing camerapositions for multiple images, result in relative differences in the 3-Dto 2-D projections of the features that are captured in the 2-D images.By determining the positions of features in 2-D images, and comparingthe relative locations of these features in images taken from differingcamera positions, the 3-D positions of the features may be determined.

However, fundamental problems still exist with current conversionmethods. For example, a typical motion picture will have a very largeand predetermined image set, which (for the purposes of camera and scenereconstruction) may contain extraneous or poorly lit images, haveinadequate variations in perspective, and contain objects with changinggeometry and image data. Nor can the known conversion methods takeadvantage of the processor saving aspects of other applications, such asrobot navigation applications that, while having to operate in real timeusing verbose and poor quality images, can limit attention to specificareas of interest and have no need to synthesize image data forsegmented objects.

In addition, existing methods of conversion are not ideally suited forscene reconstruction. The reasons for this include excessivecomputational burden, inadequate facility for scene refinement, and thepoint clouds extracted from the images do not fully expressmodel-specific geometry, such as lines and planes. The excessivecomputational burden often arises because these methods correlate all ofthe extracted features across all frames used for the reconstruction ina single step. Additionally, existing methods may not provide foradequate interactivity with a user that could leverage user knowledge ofscene content for improving the reconstruction.

The existing techniques are also not well suited to the 2-D to 3-Dconversion of things such as motion pictures. Existing techniquestypically cannot account for dynamic objects, they usually use pointclouds as models which are not adequate for rendering, and they do notaccommodate very large sets of input images. These techniques alsotypically do not accommodate varying levels of detail in scene geometry,do not allow for additional geometric constraints on object or cameramodels, do not provide a means to exploit shared geometry betweendistinct scenes (e.g., same set, different props), and do not haveinteractive refinement of a scene model.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed to systems and methods which concern2-D to 3-D images. The various embodiments of the present inventioninvolve acquiring and processing a sequence of 2-D images, generatingcamera geometry and static geometry of a scene in those usages andconverting the subsequent data into a 3-D rendering of that scene.

Embodiments of the invention are directed to systems and methods forprocessing 2-D to 3-D image conversion. Embodiments includes erodefilling when an object has been removed from an image. In oneembodiment, a target binary t image mask of the removed object is usedfor comparison with a binary source mask of the image. In an iterativeprocess, color differences between pixels in the target mask and pixelsin the source mask are identified and the pixels in the target mask arechanged working from the periphery of the target mask to the center ofthe mask. Color noise is eliminated in the filled region of the targetmask and, if desired, random color noise is added to the filled targetmask.

One embodiment of the invention is a method for erode fill whenconverting 2-D images to 3-D images, that comprises using a source maskof an image and a target mask of an image to be removed from said sourcemask, identifying all pixels in said target mask that have one or moreadjacent pixels in said source mask which are not in said target mask;and replacing all said identified pixels in said target mask with pixelshaving a color taken from neighboring pixels.

Another embodiment of the invention is a method for erode filling duringimage conversion, that comprises identifying pixels in a target maskthat have one or more adjacent pixels in a source mask which are not insaid target mask, said target mask being a binary image representingpixels that are to be removed from a source mask which is a binary imageof what is to be filled; estimating a color of each identified pixel byblending colors associated with an identified adjacent pixel; changing acolor of each said identified pixel to conform with said estimatedcolor; and iterating said identifying, estimating and changing until allpixels in said target mask have been changed.

A further embodiment of the invention is code for controlling aprocessor so as to allow said process to control erode filling duringimage conversion, that comprises control sequences for identifyingpixels in a target mask that have one or more adjacent pixels in asource mask which are not in said target mask, said target mask being abinary image representing pixels that are to be removed from a sourcemask which is a binary image of what is to be filled; control sequencesfor estimating a color of each identified pixel by blending colorsassociated with an identified adjacent pixel; control sequences forchanging a color of each said identified pixel to conform with saidestimated color; and control sequences for iterating said identifying,estimating and changing until all pixels in said target mask have beenchanged.

A further embodiment of the invention is a method for filling in animage when an object has been removed from that image, that comprisesdetermining color differences between adjacent pixels at an intersectionof a binary target mask of said removed object with pixels in a binarymask of said image; and adjusting a color of said pixels within saidtarget mask at said intersection to account for said color differences.

The foregoing has outlined rather broadly the features and technicaladvantages of the present invention in order that the detaileddescription of the invention that follows may be better understood.Additional features and advantages of the invention will be describedhereinafter which form the subject of the claims of the invention. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present invention. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the invention as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe invention, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, reference isnow made to the following descriptions taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a flow diagram illustrating the steps of 2-D to 3-D conversionaccording to an embodiment of the invention;

FIG. 2 is a flow diagram illustrating the steps for automaticallymasking an object according to one embodiment of the invention;

FIG. 3 is a flow diagram illustrating the steps of generating a camerageometry according to one embodiment of the invention;

FIG. 4 is a flow diagram illustrating the steps of managing objectocclusion according to one embodiment of the invention;

FIG. 5 is a flow diagram illustrating the steps for removing an objectand filling in the missing information according to one embodiment ofthe invention;

FIG. 6 depicts a flowchart for generating texture data according to onerepresentative embodiment;

FIG. 7 depicts a system implemented according to one representativeembodiment; and

FIG. 8 illustrates a block diagram of one embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

The process of converting a two dimensional (2-D) image to a threedimensional (3-D) image according to one embodiment of the invention canbe broken down into several general steps. FIG. 1 is a flow diagramillustrating an example process of conversion at a general level. Itshould be noted that FIG. 1 presents a simplified approach to theprocess of conversions those skilled in the art will recognize that thesteps illustrated can be modified in order such that steps can beperformed concurrently. Additionally in some embodiments the order ofsteps is dependent upon each image. For example the step of masking canbe performed, in some embodiments, up to the point that occlusiondetection occurs. Furthermore, different embodiments may not performevery process shown in FIG. 1.

Additional description of some aspects of the processes discussed belowcan be found in, U.S. Pat. No. 6,456,745, issued Sep. 24, 2002, entitledMETHOD AND APPARATUS FOR RE-SIZING AND ZOOMING IMAGES BY OPERATINGDIRECTLY ON THEIR DIGITAL TRANSFORMS, U.S. Pat. No. 6,466,205, issuedOct. 15, 2002, entitled SYSTEM AND METHOD FOR CREATING 3-D MODELS FROM2-D SEQUENTIAL IMAGE DATA, U.S. patent application Ser. No. 10/946,955,filed Sep. 22, 2004, entitled SYSTEM AND METHOD FOR PROCESSING VIDEOIMAGES, and U.S. patent application Ser. No. 11/627,414, filed Jan. 26,2007, entitled METHODOLOGY FOR 3-D SCENE RECONSTRUCTION FROM 2-D IMAGESEQUENCES, the contents of which are hereby incorporated by reference intheir entirety.

At process 101 of the illustrated embodiments ingestion of the imagesoccurs. At this step, images are digitized and brought into theassociated system for conversion. Preprocessing of the images occurs atprocess 101 to break the footage into discreet segments or cuts. Also atthis step, the footage is formed into the file structure required forthe system. Further object lists representing objects in the image arecreated for each of the discreet cuts. Additionally, other imageprocessing techniques can be performed such as color correction, thecreation of additional sequences with different file structures fordifferent parts of the process (e.g. different color depths). Cachefiles also can be created at different resolutions, and metadata alsocan be associated with the various areas of the structure.

At process 102 of the illustrated embodiment, the masking processoccurs. At this step outlines are created for the various elements ofthe individual cuts identified at process 101. During this process bothmanual and automatic processes can be used to define discreet elementsin the cuts. Again it should be noted that process 102 can occur priorto process 103, but in other embodiments process 102 can be performed atany point prior to process 106. Various masking processes arc disused inthe masking section below.

At process 103 camera reconstruction is performed. At this step camerasare created for each of the individual cuts. The cameras are createdusing various camera reconstruction techniques. From the camerareconstruction a camera file is generated for each cut. In someembodiments, camera geometry and static geometry can also be created viaartist interpretation of the ingested image sequence. Various methods ofgenerating camera and static geometry are discussed in the sectionbelow.

At process 104 of the illustrated embodiment, static modeling isperformed. At this step models of elements that are stationary withinthe scene are preferably created. Geometries representing the shape andsize are created. This conceptually forms the environment in which thecut occurs.

At process 105 of the illustrated embodiment, dynamic modeling occurs.At this step models are created for objects that are moving in theenvironment created in process 104. Geometries representing the shape,size and position of the moving objects are created relative to theenvironment and other dynamic objects in the images.

At process 106 of the illustrated embodiment, occlusion detection isperformed. During occlusion detection areas of overlap between objectsand the environment are preferably calculated. Bitmaps may then becreated to represent the occlusions. This allows for the system to knowthe discreet elements of each object that are both visible and notvisible from the camera perspective. Various methods of handlingocclusion are described in the section Handling Occlusion below.

At process 107 of the illustrated embodiment, filling is performed.Filling recreates the occluded regions of each element. This process canuse both automatic and manual techniques. A temporal filling method canbe used to recreate information in one frame that is available in otherframes from the movement of models and the camera. A spatial method canbe uses which use information available within the image itself tosynthesize the missing information. Various methods are furtherdescribed in the section Spatial Methods below.

At process 108 of the illustrated embodiment, texture generation isperformed. Texture generation takes information created at process 107and applies it to the models to create texture. Texture generation mayalso use information from images. Additional discussion of texturegeneration appears in the Texture section below.

At process 109 of the illustrated embodiment, visualization or stereovisualization is performed. This process determines the stereoscopicparameters of the image in 3-D. The point upon which the eyes or thecamera converge, the convergence point, is defined as is theinter-ocular distance, the space between the eyes or the cameras. Thesevalues are applied either as static or dynamic values. Visualization isfurther described in the Visualization section below.

The final process of the conversion as set forth in the illustratedembodiment is the rendering of the image. This is shown at process 110.At 110 based on the result of process 109 full resolution of the imagesis rendered for the left and right images. Rendering further describedin the Rendering section below. Note that in its most minimal form,embodiments define a single object, set the stereo parameters, andrender.

Masking

In one embodiment, a mask is generated from an object model using apolycurve. A polycurve is a closed sequence of adjacent parametriccurves, such as Bezier or B-spline curves. However, in other embodimentsthe polycurve, or other structure, is a definition of a subset of pixelsin each image representing an object. A 2-D polycurve can be taken torepresent a planar region under certain geometric constraints (e.g., noconstituent curve intersects any other except at endpoints). A 2-Dpolycurve (exterior) together with zero or more 2-D polycurves (holes)is taken to represent a planar region with holes, provided all of thehole regions are contained within the exterior and are disjoint. A 2-Dmesh representation of this region is constructed by overlaying theregion with a rectangular grid of a chosen resolution. Each grid elementis classified as being interior to the region, exterior to the region,or intersecting the boundary of the region. In one embodiment, a mesh isgenerated through a modified scan-line traversal of the grid, where thenumber of triangles contributed by each grid element to the resultingmesh depends on its classification. In this classification, exteriorelements have no contribution, interior elements contribute twotriangles, and boundary elements contribute one or more trianglesdepending on how many corners of the grid element lie within the region.

The resulting 2-D mesh is converted into a 3-D mesh byreverse-projecting it onto any viewing plane in the 3-D space. Further,the obtained 3-D planar mesh is ‘lofted’ to form a 3-D mesh with volumeby replicating/mirroring the surface structure. In one embodiment, thisprocess introduces additional copies of internal vertices or of allvertices together with the required edges and faces. The resulting 3-Dmesh can then be manipulated by any one of the standard modelingtechniques known to those of skill in the art.

In an alternative embodiment, the 3-D mesh is converted to 2-Dpolycurves. Conversion from a 3-D mesh to a collection of 2-D polycurveswhich define a region is achieved by first rendering the 3-D mesh toproduce a boolean image. Next, using image processing, partitioningoccurs to partition the boundary pixels into a collection of pixel pathsrepresenting an exterior and a list of holes. Then curve fittingtechniques are used to generate polycurves which approximate each ofthese component regions.

There have been numerous attempts to use auto-masks to either fullygenerate a mask or to refine an initially supplied mask, Livewire andLivelane are techniques that assist a user in masking to increaseefficiency and accuracy. These have been discussed in “User-steeredimage segmentation paradigms: live wire and live lane” Graphical Modelsand Image Processing, Volume 60, Issue 4 (July 1998) Pages: 233-260, thecontents of which are hereby incorporated by reference in theirentirety. In general, these techniques utilize the fact that a user isgood at recognizing objects and can quickly provide a crude mask,whereas a computer is good at calculating the optimal pixels thatseparate the object.

Active contours and snakes are other techniques that can either be userassisted or fully automated. Active contours have been described in IEEETransactions on Volume 10, Issue 2, February 2001 Page(s):266-277, thecontents of which are hereby incorporated by reference in theirentirety. In general, these techniques utilize a curve and iterativelyevolve the curve, subject to a set of constraints, until an equationminimum has been found. The goal is that the equation will be at aminimum when the curve matches the outline of one or more objects.

One embodiment of the present disclosure is directed to a novelfeature-tracking algorithm for use in generating a mask. This approachis based off of the data structures used to represent the mask. Thepolycurve polygon, a series of polycurves connected to form a loop, isthe primary data structure used to define a mask. The feature trackingalgorithm assumes that every endpoint of a polycurve corresponds to afeature in the image. The algorithm is preferably initialized byproviding a very accurate polycurve polygon of the desired object to bemasked in a very small subset of the images. In one embodiment, thesubset used represents an equal distribution of the total set, such asthe first, middle, and last image. The feature tracker algorithm in suchan embodiment can then track all the endpoint vertices of the polycurvepolygon. The algorithm assumes that the object is more or less rigid,and thus the position of the control vertices of polycurve polygonremain relative to endpoint vertices. In some embodiments the controlvertices can be automatically refined using image analysis tools.

One current embodiment of the feature tracker algorithm of the presentinvention works in a sequential manner. That is, the feature trackingalgorithm starts at either end of the image set and works its way to theother. The algorithm can work on the image set in any order. Thealgorithm assumes not only that the provided frames are correct but alsoall previously tracked frames are correct and uses that information tohelp locate the position of the vertex in subsequent frames. Thus, ifthe tracker algorithm fails to locate the correct position of a vertexin one frame, it will likely fail to locate the correct position in many(or all) subsequent frames. In this event the user is able to manuallyspecify the position of the vertex in any of the incorrect frames. Thealgorithm is then rerun with the assumption that this newly providedinformation is accurate and the additional information will besufficient to track the vertex in all other incorrect frames. Thisiterative process can be performed as many times as necessary tocorrectly track all features and thus correctly mask the object.

In another embodiment, an alternative approach for masking is used. Inthis approach, masks, feature detection, and tracking, together withpossible user input, are used as the basis for deriving a set of‘features’ within a sequence of images. Each such feature comprises alist of pairs: the first element indicating the source image and thesecond element describing the feature geometry in image coordinates(e.g. a 2D point). Such a feature set is described in relation to amethod of camera reconstruction described in U.S. patent applicationSer. No. 11/627,414 (hereby incorporated herein by reference), whereeach feature determines a point in 3D scene coordinates. Thereconstruction process determines this 3D point through a globaloptimization process according to which an error value is assigned eachfeature to describe how closely the estimated camera model maps the 3Dpoint to the observed 2D image points. A threshold on this error valuecan be used to recognize ‘outliers’—features whose inclusion in theoptimization process has a detrimental effect on the calculated scenegeometry. This provides a means of classifying features as static vsdynamic: features whose image-coordinate motion is accurately describedby the camera model correspond to static geometry, while thecomplementary set correspond to dynamic geometry.

Given an initial classification of features into static/dynamic, camerareconstruction can be used in an iterative fashion to refine thisclassification. The initial classification can be given in the form ofapproximate boolean image masks for the dynamic objects. The initialsegmentation can be used to determine weights on features for use in theoptimization process—static features receiving higher weights thandynamic features and thus having the greater influence on the calculatedscene geometry. Each iteration of the reconstruction process preferablyupdates the classification according to the determined outliers andadjusts the feature weights for the next iteration. The iterationprocess terminates when either: the feature classification does notchange; a specified condition on the global error has been achieved; ora specified number of iterations has been performed. A final iterationshould be performed without the influence of dynamic features (e.g. withzero weights) to ensure the accuracy of the calculated camera and staticgeometry.

The feature classification resulting from the iterative camerareconstruction process can be used to refine the initially suppliedapproximate dynamic object masks. If the initial masks were not suppliedas polycurve regions, then such can be generated from boolean images.These initial polycurve regions can then be refined so as to include alldynamic features and to exclude all static features. A static featuremay be excluded from a polycurve region by determining the nearestcomponent curve and adjusting its parameters algorithmically. A dynamicfeature can be included by determining the nearest object region and thenearest component curve therein, and adjusting the curve parametersalgorithmically. This process may utilize user interaction to obtainoptimal results in some cases.

A third embodiment is directed to auto generating the mask from a pointcloud. During the camera reconstruction process and triangulationprocess a great deal of 3-D features can be produced in the form of apoint cloud. The point cloud may be grouped using a segmentationprocess. In this approach a group of vertices represents an object inthe scene. The segmentation process can be user assisted or completelymanual. Following segmentation a group of feature points and the cameraare used to generate a mask. In some embodiments, this mask may be verycrude, but in these instances another technique, such as the aforementioned techniques can be used to automatically refine the mask.

Segmentation can also be fully automated. The point cloud (as discussedbelow) can be rendered to a Boolean image. This can occur via creating amesh from the point cloud, or by rendering each point and calculatingthe outline of the object by calculating the corresponding convex hullor using the previously mentioned active contours. From this Booleanimage a polycurve can be create via segmentation and curve fittingtechniques. This polycurve could then be refined manually, automaticallyor manually assisted. Refining may include image processing techniquessuch as the previously listed livewire, active contours, or featuretracking. Segmentation has been discussed in for example: “Recognisingstructure in lascr scanner point clouds,” G. Vosselman, B. G. H. Gorte,G. Sithole and T. Rabbanim, published by IAPRS and found atwww.itc.nl/personal/vosselman/papers/vosselman2004.natscan.pdf: “Shapesegmentation and matching with flow discretization,” T. K. Dey, J.Giesen and S. Goswami. Proc. Workshop Algorithms Data Structures (WADS03), LNCS 2748, F. Dehne, J.-R. Sack, M. Smid Eds., 25-36; and “Shapesegmentation and matching from noisy point clouds,” T. K. Dey, 3. Giesenand S. Goswami, Proc. Eurographics Sympos. Point-Based Graphics (2004),Marc Alexa and S. Rusinkiewicz (eds) (2004), 193-199, the contents ofwhich are hereby incorporated by reference in their entirety.

A mask (i.e., a 2-D polycurve) is obtained in one embodiment from a 3-Dpoint cloud according to the process illustrated in FIG. 2. A pointcloud is rendered according to the camera model to obtain a booleanimage of process 201 of the illustrated embodiments of the illustratedembodiments. Next a scan-line algorithm may be employed to identify asequence of boundary points, as is illustrated at process 202. Curvefitting techniques may be used to obtain a polycurve at process 203. Theresulting polycurve is refined at process 204. This refinement can bedone manually by a user, or automatically in conjunction with thepreviously described contour-based methods. Method of refinement hasbeen discussed in “Reconstructing B-spline Curves from Point Clouds—ATangential Flow Approach Using Least Squares Minimization,” Yang Liu,Huaiping Yang, and Wenping Wang, International Conference on ShapeModeling and Applications 2005 (SMI'05) pp. 4-12, the contents of whichare incorporated by reference in their entirety.

Camera and Static Geometry

In one embodiment, 3-D camera and scene geometry are used to calculatenew 3-D geometry from supplied 2-D information via triangulation.Triangulation is discussed in U.S. patent application Ser. No.11/627,414, filed Jan. 26, 2007, entitled “Methodology For 3-D SceneReconstruction From 2-D Image Sequences” the contents of which arehereby incorporated by reference in their entirety. Briefly, paragraph38 of the referenced Application is reproduced below.

The addition of mesh detail through triangulation is performed,according to some embodiments, involves adding structure to the mesh(i.e., vertices, edges and faces), and then triangulating the locationof each new vertex with reference to a set of frames for which camerageometry has already been calculated (e.g., key frames). The newvertices may be assigned to images at each of the selected frames. Thenthe underlying 3-D scene location is calculated through triangulation.This assignment of image coordinates can be performed, in someembodiments, through user input or through application of automatedfeature detection and tracking algorithms. It should be noted that themore frames that are providing observed image coordinates for a vertex,the greater the accuracy of triangulated scene-coordinate point.

2-D images capture a great deal of information that can be used in awide variety of ways depending on the viewer. Unfortunately, there isalso a great deal of information that 2-D images cannot capture that isuseful. Most notably is the loss of 3-D information. Viewers have theability to infer relative positions within a 2-D image, and givencertain camera angles can even take measurements from the 2-Dinformation. However, there are often times when 3-D information of thescene or features within the scene would be of great use.

Several techniques are used to generate 3-D scene information that isassociated with a set of 2-D images. Examples of this informationinclude reconstruction from a sequence of 2-D images, on sitemeasurements from tools such as Global Positioning System (GPS) orlasers, and artistic creation with 3-D software with or without the aidof 2-D images. Some of these techniques, such as artistic creation, canproduce data that encompasses all of the 3-D information of a scene.However, most only create a small subset of the available 3-Dinformation.

In one embodiment, using the provided image sequence and cameraparameters that were used to create the image sequence, new 3-D geometrycan be calculated. It should be noted that although existing scenegeometry is not required, in some embodiments it is provided andenhances the usefulness of new geometry. A specific feature iscalculated provided that a sufficient number (2 or more) of images inthe sequence contain that feature, and that the cameras used to capturethe images have sufficient disparity.

FIG. 3 is a flow diagram illustrating the process for generating a 3-Dgeometry according to one embodiment. A subset of the images containingthe feature is chosen either as input from a user or automaticallycalculated. This is shown at process 301. 2-D vertex positions areprovided, at process 302. These vertex positions represents the imagecoordinates corresponding to the feature in each of the images in thechosen subset. The cameras representing each image of the subset areused to triangulate the 3-D position that best describes all of the 2-Dvertex positions. This is shown at process 303. It should be noted thatthere are several factors that effect the accuracy of the calculated 3-Dposition. These factors include the accuracy of the camera parameters,the disparity of the cameras, the accuracy in the 2-D vertex positionsand the number of images in the subset.

The above process can be automated in numerous ways. For example,feature detection locates features that stand out in the image sequence.Feature tracking can position 2-D vertices in subsequent imagesprovided, however, that an initial 2-D vertex is placed in the firstimage. These two techniques are coupled to automatically generatefeatures and their associated 2-D image coordinates. Given a set imagesand 2-D vertex information a subset of images and corresponding 2-Dvertices are automatically selected to produce the greatest accuracy.

Handling Occlusion

An occlusion is a the portion of an objects surface which is not visiblefrom a given perspective or camera configuration. This obscuring ofportions of the object can occur for a number of reasons. First,back-facing portions as well as portions outside the camera viewingvolume are obscured. Second, inter-object occlusion of front-facingportions of an object occur when other objects appear between the cameraand the object of interest. Third, intra-object occlusion offront-facing portions of an object occur when non-convex geometry of theobject obscure portions of the object. An occlusion is logically relatedto an object's texture, and may be considered as an extra channel ofboolean image data (i.e. true to indicate occlusion, false fornon-occlusion).

In one embodiment, inter-object occlusions for projective textures arecalculated according to the process illustrated in FIG. 4. At process401 the entire scene is rendered according to the camera perspective toform a depth image S. Next, the object of interest is rendered inisolation according to the same camera perspective to form a seconddepth image T, and (if necessary) a boolean image indicating the objectmask M. This is shown at process 402. Note that each of these threeimages (S, T and M) have the same dimensions. A boolean occlusion imageof the same dimension is formed where each pixel p has the value

O[p]:=if M[p] then S[p]˜<(or ≠)T[p] else false  Equation 1

This formation of the boolean occlusion image is illustrated at process403.

In one embodiment, the process of FIG. 4 is implemented in a graphicsapplication programming interface (API) using “shadow mapping”. Here thedepth image S is pre-computed and loaded as a depth texture. Then theshadow mapping functionality of the graphics API is used to calculatethe boolean occlusion image directly from a single pass render of thechosen object in isolation. In open graphics library (OpenGL), forexample, a simple fragment shader can be used to output a boolean valueby comparing the depth buffer (i.e. the object depth) to the depthtexture (i.e. the scene depth).

For the purpose of 2-D to 3-D conversion, additional flexibility isrequired when introducing inter-object occlusions. Specifically, objectmodels often overlap slightly at their boundaries, and treating suchoverlap as occlusion can introduce unnecessary filling artifacts. In oneembodiment, this problem is avoided by adding a user-controlledtolerance value to the comparison function of Equation 1. Specifically,the condition becomes:

abs(S[p]−T[p])>tolerance  Equation 2

For the purpose of optimizing 2-D to 3-D conversion, it is desirable tolimit the reconstruction of texture data to the portions of an objectsurface which are actually visible in the derived stereo pair. In oneembodiment, this is accomplished by generating additional occlusionimages OL and OR that corresponding to the left and right-eyeperspectives according to the above process. Then replacing theocclusion O with the intersection of O, OL and OR. In some embodiments,a set of stereo cameras can be used to calculate the optimal occlusion.This approach allows some flexibility in choosing the stereo cameraparameters during rendering without allowing any new occlusions toappear.

Spatial Methods

Spatial methods can be used in a plurality of filling techniques. Onesuch technique is the brush fill, which is a highly user involvedprocess. Another technique is the erode fill, which is an automaticprocess. Each technique has benefits in different situations.

The erode fill is designed to be an automatic filling process. There area number of ways in which an object can be specified for removal. Thiscan be done using the manual or automatic masking techniques discussedabove. Further, for example, outlining the object via manualrotoscoping, using an automatic selection tool such as a wand, or bymanually selecting all the pixels that pertain to the object. Afterremoving an object the remaining objects should be modified to accountfor the removed object. Ideally, the remaining image should look asnatural as possible, that is it should appear to a viewer as if theremoved object was never in the image. Thus, to remove the object all ofthe pixels of the object should be changed to represent what is behindthe object. Currently there are several techniques to achieve this.These include manually painting, temporal filling, or brushes. Manytimes what is behind the object is quite simple, like a white wall orblue sky. Thus, it is desirable to have an automatic filling process tofill in this simple information.

In one embodiment, the system takes an image and a binary image (themask) representing the pixels that are to be removed as a startingpoint. The process of the present embodiment replaces those pixels withinformation taken from the remaining image. The process is iterative,and can require a very large number of iterations to complete theprocess. First the process identifies all pixels in the mask that haveone or more adjacent pixels that are not in the mask. This is shown inprocess 501 of FIG. 5. Next the process estimates the color of each ofthese pixels by blending the color of the adjacent pixels. Each of thesefilled pixels is then removed from the mask, at process 503, andprocesses 501 and 503 are repeated again. Essentially then the color ofeach of the pixels in the target mask is changed to match thesurrounding pixel color, starting from the periphery and working inward.After each iteration pixels are essentially removed (color changed) fromthe mask so that as the process works toward the center of the targetmask new pixels will be found with at least one adjacent pixel not inthe unfilled mask. Thus, after each iteration the mask becomes smalleruntil there are no pixels left unchanged in the mask. Note that the flowof FIG. 5 is an example, and other embodiments may use fewer processes.

However, sometimes it does not make sense to assume that all pixels notin the provided mask can be used. Thus, in some embodiments, a secondbinary image can be provided representing the pixels that the processcan choose from. Thus, for each iteration only pixels with at least oneadjacent pixel from the second supplied mask will be filled at process503. In this embodiment, processes of FIG. 5 terminate when theintersection of the source and target masks is empty.

In another embodiment, the processes depicted in FIG. 5 accounts fordigital noise that is found in most digital images. If during thefilling and estimating process a noise pixel is adjacent to a pixel tobe filled, and it is the only adjacent pixel then the color of thisnoise will be copied to the filled pixel. In order to avoid a copying ofnoise, in one embodiment, only pixels with a sufficient number ofadjacent good pixels (i.e. pixels having substantially the same color)will be filled. This approach causes the noise pixels to be blended withnon-noise pixels giving the filled pixel a much more accurate color. Inone embodiment, the sufficient number of pixels can be supplied by theuser, or can be automatically determined via analysis of the suppliedimage. To prevent the resulting filled image from appearing too smooth,because there is noise throughout the image except in area that has beenfilled, one embodiment provides noise to the filled image either duringthe filling process or via separate process used in conjunction with thefilling process. This is shown at optional process 505.

Texture Data

FIG. 6 is a flowchart depicting one example embodiment for creatingtexture map data for a three dimensional object for a particulartemporal position. The flowchart for creating texture map data begins instep 601 of the depicted embodiment where a video frame is selected. Theselected video frame identifies the temporal position for which thetexture map generation will occur. In step 602 of the depictedembodiment, an object from the selected video frame is selected.

In step 603, of the depicted embodiment, surface positions of the threedimensional model that correspond to visible portions of the selectedobject in the selected frame are identified. The identification of thevisible surface positions may be performed, as an example, by employingray tracing from the original camera position to positions on the threedimensional model using the camera reconstruction data. In step 604 ofthe depicted embodiment, texture map data is created from image data inthe selected frame for the identified portions of the three dimensionalmodel.

In step 605 of the depicted embodiment, surface positions of the threedimensional model that correspond to portions of the object that werenot originally visible in the selected frame are identified. In oneembodiment, the entire remaining surface positions are identified instep 605 thereby causing as much texture map data to be created for theselected frame as possible. In certain situations, it may be desirableto limit construction of the texture data. For example, if texture datais generated on demand, it may be desirable to only identify surfacepositions in this step (i) that correspond to portions of the object notoriginally visible in the selected frame and (ii) that have becomevisible due to rendering the object according to a modification in theviewpoint. In this case, the amount of the object surface exposed due tothe perspective change can be calculated from the object's cameradistance and a maximum inter-ocular constant.

In step 606 of the depicted embodiment, the surface positions identifiedin step 605 are correlated to image data in frames prior to and/orsubsequent to the selected frame using the defined model of the object,object transformations and translations, and camera reconstruction data.In step 607 of the depicted embodiment, the image data from the otherframes is subjected to processing according to the transformations,translations, and camera reconstruction data. For example, if a scalingtransformation occurred between frames, the image data in the prior orsubject frame may be either enlarged or reduced depending upon thescaling factor. Other suitable processing may occur. In onerepresentative embodiment, weighted average processing may be useddepending upon how close in the temporal domain the correlated imagedata is to the selected frame. For example, lighting characteristics maychange between frames. The weighted averaging may cause darker pixels tobe lightened to match the lighting levels in the selected frame. In onerepresentative embodiment, light sources are also modeled as objects.When models are created for light sources, lighting effects associatedwith the modeled objects may be removed from the generated textures. Thelighting effects would then be reintroduced during rendering.

In step 608 of the depicted embodiment, texture map data is created forthe surface positions identified in step 605 from the data processed instep 607 of the depicted embodiment. Because the translations,transformations, and other suitable information are used in the imagedata processing, the texture mapping of image data from other framesonto the three dimensional models occurs in a relatively accuratemanner. Specifically, significant discontinuities and other imagingartifacts generally will not be observable.

In one representative embodiment, steps 604-607 are implemented inassociation with generating texture data structures that represent thesurface characteristics of an object of interest. A given set of texturedata structures define all of the surface characteristics of an objectthat may be recovered from a video sequence. Also, because the surfacecharacteristics may vary over time, a texture data structure may beassigned for each relevant frame. Accordingly, the texture datastructures may be considered to capture video information related to aparticular object.

Visualization

Another embodiment of the present disclosure is directed to aninteractive system for observing and manipulating the stereo effect in asequence of stereo pairs (frames) obtained by rendering a scene modelaccording to a stereo camera model. The stereo camera model comprises agiven camera model together with values for inter-ocular andconvergence-point distances suitably animated over a sequence of frames.The scene model comprises a number of object models together with anyadditional data required to render realistic images of the scene at thedesired frames (e.g. lighting data). Each object model includes the datarequired to render realistic images of the object at the desired frames.This data may comprise a mesh specifying the object geometry to asufficient level of detail, texture data, and masking data used to clipthe projection of approximate geometry.

For the purpose of 2-D to 3-D conversion, in one embodiment, theoriginal image sequence serves uniformly as the texture data for allobjects using the techniques of projective texture mapping.

Embodiments allow for the independent choice of both an inter-oculardistance and a convergence distance for each of the desired frames.These parameters can be specified using a Graphical User Interface(GUI), or through an input device, such as a keyboard or mouse.Embodiments allow users to move freely through the sequence of framesadjusting the stereo camera parameters, and observing the correspondingeffects through any viable means for stereo pair presentation (e.g.interleaved via shutter glasses, dual projector via polarization). Usinginterpolation techniques, these values are preferably specified atchosen key frames to reduce the burden on the user. this allows the userto define any number of stereo camera models, and to switch betweenthese modes easily for reference. In some embodiments, caching ofrendered images is used to improve performance when switching betweenframes.

This process allows for the rendering of the scene in its entirety, orto limit rendering to a selected collection of objects within the scene.This allows for rendering objects as wireframes, as surfaces withtexture data, in some embodiments, or permit any other visual effectapplied (e.g. lighting). In the case of texture mapping, in someembodiments, the technique applied may be a simplification of thetechnique use in final rendering. That is the process may not use depthpeeling.

Rendering

In one embodiment, rendering of images is performed by depth peeling.Depth peeling is an order independent approach that allows for correctblending of inter related models. Typically rendering engines either usea depth traversal tree, rendering layers or some sort of per pixelrendering algorithm (ray tracing, global illumination, photon mapping,radiosity, etc . . . ). The rendering approach of the presentembodiments is capable of generating both left and right eye images fromscene assets using the stereoscopic variables as defined within thestereoscopic camera data structure. The left and right-eye projectionsfor stereo rendering are obtained with respect to the original cameramodel using an “off-axis” method. To achieve an order independentrendering engine a multi pass approach is taken. For each image that iscreated a number of scene traversals are performed that peel off eachdepth layer based on a previous traversals depth map. After completionof rendering all of the layers, the layers are then blended togetherfrom back to front. The rendering engine of the present embodiments canrender out the scene in its entirety or just in portions. Further therendering engine can also change the number of rendering traversals.Blending of the layers is done on the final stage, and can use any knownblend function to generate the correct blending. To increase smoothnessamong objects edges anti-aliasing can be performed to remove anyunwanted jaggies (jagged edges).

Implementation

An embodiment of the present invention may utilize a general purposeprocessor-based system, such as PC 700 illustrated in FIG. 7, adapted tomanipulate video image information through the use of machine visionalgorithm 710 and video generation algorithm 720. PC 700 includesprocessor (CPU) 701 coupled to memory (RAM) 702. RAM 702 providesstorage for digitized image information associated with a source videoimage as well as for any video image resulting from the operation of thepresent invention. PC 700 is also adapted to accept input of a sourcevideo as well as output a resulting video. Of course, acceptance andoutput of such video may be in digitized form. Alternatively, PC 700 maybe adapted to accept and/or output analogue video, such as in the formof National Television System Committee (NTSC) compatible signals. Itshould be noted that while a processor is shown, the system could behard wires, or could be a series of processors.

PC 700 also includes an operator interface providing informationexchange with an operator of the system. Such information exchange mayinclude the display of source and/or resulting video images on asuitable display device. Additionally, the information exchange mayinclude an operator selecting and/or inputting information with respectto the generation of video images according to the present invention.

FIG. 8 depicts system 800 for processing a sequence of video imagesaccording to one representative embodiment. System 800 may beimplemented on a suitable computer platform such as depicted in FIG. 7.System 800 includes conventional computing resources such as centralprocessing unit 801, random access memory (RAM) 802, read only memory(ROM) 803, user-peripherals (e.g., keyboard, mouse, etc.) 804, anddisplay 805. System 800 further includes non-volatile storage 806.

Non-volatile storage 806 comprises data structures and software code orinstructions that enable conventional processing resources to implementsome representative embodiments. The data structures and code mayimplement the flowcharts of FIGS. 6 and 7 as examples.

As shown in FIG. 8, non-volatile storage 806 comprises video sequence807. Video sequence 807 may be obtained in digital form from anothersuitable medium (not shown). Alternatively, video sequence 807 may beobtained after analog-to-digital conversation of an analog video signalfrom an imaging device (e.g., a video cassette player or video camera).Object matting module 814 defines outlines of selected objects using asuitable image processing algorithm or algorithms and user input. Camerareconstruction algorithm 817 processes video sequence 807 to determinethe relationship between objects in video sequence 807 and the cameraused to capture the images. Camera reconstruction algorithm 817 storesthe data in camera reconstruction data 811.

Model selection module 815 enables model templates from model library810 to be associated with objects in video sequence 807. The selectionof models for objects are stored in object models 808. Object refinementmodule 816 generates and encodes transformation data within objectmodels 808 in video sequence 807 using user input and autonomousalgorithms. Object models 808 may represent an animated geometryencoding shape, transformation, and position data over time. Objectmodels 808 may be hierarchical and may have an associated template type(e.g., a chair).

Texture map generation module 821 generates textures that represent thesurface characteristics of objects in video sequence 807. Texture mapgeneration module 821 uses object models 808 and camera data 811 togenerate texture map data structures 809. Preferably, each objectcomprises a texture map for each key frame that depicts as much surfacecharacteristics as possible given the number of perspectives in videosequence 807 of the objects and the occlusions of the objects. Inparticular, texture map generation module 821 performs searches in priorframes and/or subsequent frames to obtain surface characteristic datathat is not present in a current frame. The translation and transformdata is used to place the surface characteristics from the other framesin the appropriate portions of texture map data structures 809. Also,the transform data may be used to scale, morph, or otherwise process thedata from the other frames so that the processed data matches thecharacteristics of the texture data obtained from the current frame.Texture refinement module 822 may be used to perform user editing of thegenerated textures if desired.

Scene editing module 818 enables the user to define how processed imagedata 820 is to be created. For example, the user may define how the leftand right perspectives are to be defined for stereoscopic images if athree dimensional effect is desired. Alternatively, the user may providesuitable input to create a two dimensional video sequence having otherimage processing effects if desired. Object insertion and removal mayoccur through the receipt of user input to identify objects to beinserted and/or removed and the frames for these effects. Additionally,the user may change object positions.

When the user finishes inputting data via scene editing module 818, theuser may employ rendering algorithm 819 to generate processed image data820. Processed image data 820 is constructed using object models 808,texture map data structures 809, and other suitable information toprovide the desired image processing effects.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

1. A method for generating image information when converting 2-D imagesto 3-D images, said method comprising: using a source mask of an imageand a target mask of an image to be removed from said source mask,identifying all pixels in said target mask that have one or moreadjacent pixels in said source mask which are not in said target mask;and replacing all said identified pixels in said target mask with pixelshaving a color taken from neighboring pixels.
 2. The method of claim 1wherein said identifying comprises: determining pixels in said maskhaving one or more adjacent pixels not in said mask.
 3. The method ofclaim 2 wherein said replacing comprises: estimating a color of eachdetermined pixel by blending colors associated with said neighboringpixels; and substituting each said determined pixel with a pixel havingsaid estimated color.
 4. The method of claim 1, wherein the source maskis the negation of the target mask.
 5. The method of claim 3, whereinthe neighboring pixels are located within a selectable tolerance of thedetermined pixel.
 6. The method of claim 5, wherein the selectabletolerance is selected by one of manual setting or automatic setting. 7.The method of claim 1, further comprising: using the target mask of animage, wherein the target mask denotes pixels to be filled of saidimage.
 8. The method of claim 3 further comprising; iterating saiddetermining, estimating and substituting until all pixels in said targetmask have been substituted.
 9. The method of claim 8 further comprising:selecting a second binary mask; and wherein said determining is withrespect to said second mask for at least one of said iterations.
 10. Themethod of claim 9 further comprising ending said iteration when anintersection of said source mask and said target masks is empty.
 11. Themethod of claim 8 wherein said estimating comprises: eliminating colornoise from said estimating.
 12. The method of claim 11 furthercomprising: selectively adding color noise to said substituted pixels.13. A method for generating image information during image conversion,said method comprising: identifying pixels in a target mask that haveone or more adjacent pixels in a source mask which are not in saidtarget mask, said target mask being a binary image representing pixelsthat are to be removed from a source mask which is a binary image ofwhat is to be filled; estimating a color of each identified pixel byblending colors associated with an identified adjacent pixel; changing acolor of each said identified pixel to conform with said estimatedcolor; and iterating said identifying, estimating and changing until allpixels in said target mask have been changed.
 14. The method of claim 13further comprising: selecting a second target mask; and wherein saididentifying for at least some of said pixels is with respect to saidsecond target mask for at least one of said iterations.
 15. The methodof claim 14 further comprising ending said iterations when anintersection of said source mask and said target masks is empty.
 16. Themethod of claim 13 wherein said estimating comprises: eliminating colornoise from said estimating.
 17. The method of claim 8 wherein saideliminating comprises: using only adjacent pixels where more than acertain number of pixels have substantially a same color.
 18. The methodof claim 13 further comprising: selectively adding color noise to saidchanged pixels.
 19. Code for controlling a processor so as to allow saidprocess to control erode filling during image conversion, said codecomprising: control sequences for identifying pixels in a target maskthat have one or more adjacent pixels in a source mask which are not insaid target mask, said target mask being a binary image representingpixels that are to be removed from a source mask which is a binary imageof what is to be filled; control sequences for estimating a color ofeach identified pixel by blending colors associated with an identifiedadjacent pixel; control sequences for changing a color of each saididentified pixel to conform with said estimated color; and controlsequences for iterating said identifying, estimating and changing untilall pixels in said target mask have been changed.
 20. The code of claim19 further comprising: control sequences for selecting a second targetmask; and wherein said identifying for at least some of said pixels iswith respect to said second mask for at least one of said iterations.21. The code of claim 20 further comprising control sequences for endingsaid iterations when an intersection of said source mask and said targetmasks is empty.
 22. The code of claim 19 wherein said control sequencesfor estimating comprises: control sequences for eliminating color noisefrom said estimating.
 23. The code of claim 22 wherein said controlsequences for eliminating comprises: control sequences for selectingonly adjacent pixels where more than a certain number of pixels havesubstantially a same color.
 24. The code of claim 23 further comprising:control sequences for selectively adding color noise to said changedpixels.
 25. A method for filling in an image when an object has beenremoved from that image, said method comprising: determining colordifferences between adjacent pixels at an intersection of a binarytarget mask of said removed object with pixels in a binary mask of saidimage; and adjusting a color of said pixels within said target mask atsaid intersection to account for said color differences.
 26. The methodof claim 25 further comprising: repeatedly repeating said determiningfor intersections of unadjusted pixels within said target mask withadjacent ones of said adjusted pixels until all pixels within saidtarget mask have been adjusted.
 27. The method of claim 25 wherein saidadjusting comprises: eliminating color noise from said determined colordifferences.
 28. The method of claim 27 wherein said eliminatingcomprises: using only adjacent pixels where more than a certain numberof pixels have substantially a same color.
 29. The method of claim 25further comprising: selectively adding color noise to said adjustedpixels.